Radar-Enabled Sensor Fusion

ABSTRACT

This document describes apparatuses and techniques for radar-enabled sensor fusion. In some aspects, a radar field is provided and reflection signals that correspond to a target in the radar field are received. The reflection signals are transformed to provide radar data, from which a radar feature indicating a physical characteristic of the target is extracted. Based on the radar features, a sensor is activated to provide supplemental sensor data associated with the physical characteristic. The radar feature is then augmented with the supplemental sensor data to enhance the radar feature, such as by increasing an accuracy or resolution of the radar feature. By so doing, performance of sensor-based applications, which rely on the enhanced radar features, can be improved.

PRIORITY

This application claims priority to U.S. Non-Provisional patentapplication Ser. No. 15/286,512 filed on Oct. 5, 2016, which, in turn,claims priority to U.S. Provisional Patent Application Ser. No.62/237,975 filed on Oct. 6, 2015, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND

Many computing devices and electronic devices include sensors to providea seamless and intuitive user experience based on a device'ssurroundings. For example, a device may exit a sleep state responsive toan accelerometer indicating device movement or a touch screen of thedevice can be disabled responsive to a proximity sensor that indicatesproximity with the user's face. Most of these sensors, however, havelimited accuracy, range, or functionality, and are only able to sense acoarse or drastic change of the device's surroundings. Thus, withoutaccurate sensor input, the device is often left to infer different typesof user interaction or whether the user is even present, which resultsin incorrect user input, false or non-detection of the user, and userfrustration.

Examples of sensor inaccuracy in the above context include a device thatincorrectly exits the sleep state responsive to an accelerometer sensingnon-user-related movement (e.g., a moving vehicle) and disabling a touchscreen in response to a user holding a device incorrectly and partiallyobstructing the proximity sensor. In such cases, a device's battery canbe run down due to inadvertent power state transitions and user inputthrough the touch screen is disrupted until the user moves his handThese are just a few examples of sensor inaccuracy that can disrupt theuser's interactive experience with the device.

SUMMARY

This disclosure describes apparatuses and techniques for radar-enabledsensor fusion. In some embodiments, a radar field is provided andreflection signals that correspond to a target in the radar field arereceived. The reflection signals are transformed to provide radar data,from which a radar feature indicating a physical characteristic of thetarget is extracted. Based on the radar features, a sensor is activatedto provide supplemental sensor data associated with the physicalcharacteristic. The radar feature is then augmented with thesupplemental sensor data to enhance the radar feature, such as byincreasing an accuracy or resolution of the radar feature. By so doing,performance of sensor-based applications, which rely on the enhancedradar features, can be improved

In other aspects, a radar sensor of a device is activated to obtainradar data for a space of interest. Three-dimensional (3D) radarfeatures are extracted from the radar data and positional data isreceived from sensors. Based on the positional data, spatial relation ofthe 3D radar features is determined to generate a set of 3D landmarksfor the space. This set of 3D landmarks is compared with known 3Dcontext models to identify a 3D context model that matches the 3Dlandmarks. Based on the matching 3D context model, a context for thespace is retrieved and used to configure contextual settings of thedevice.

This summary is provided to introduce simplified concepts concerningradar-enabled sensor fusion, which is further described below in theDetailed Description. This summary is not intended to identify essentialfeatures of the claimed subject matter, nor is it intended for use indetermining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of radar-enabled sensor fusion are described with referenceto the following drawings. The same numbers are used throughout thedrawings to reference like features and components:

FIG. 1 illustrates an example environment that includes a computingdevice having a radar sensor and additional sensors.

FIG. 2 illustrates example types and configurations of the sensors shownin FIG. 1.

FIG. 3 illustrates example implementations of the radar sensor shown inFIG. 1 and corresponding radar fields.

FIG. 4 illustrates another example implementation of the radar sensorshown in FIG. 1 and a penetrating radar field.

FIG. 5 illustrates an example of configuration of components capable ofimplementing radar-enabled sensor fusion.

FIG. 6 illustrates an example method for augmenting radar data withsupplemental sensor data.

FIG. 7 illustrates an example of implementation of motion tracking withenhanced radar features.

FIG. 8 illustrates an example method for low-power sensor fusion inaccordance with one or more embodiments.

FIG. 9 illustrates an example of low-power sensor fusion implemented bysmart-television that includes a sensor fusion engine.

FIG. 10 illustrates an example method for verifying a radar feature withcomplimentary sensor data.

FIG. 11 illustrates an example method for generating a context model fora space of interest.

FIG. 12 illustrates an example of a room being contextually mapped inaccordance with one or more embodiments.

FIG. 13 illustrates an example method for configuring context settingsbased on a context associated with a space

FIG. 14 illustrates an example method of changing contextual settings inresponse to a context of a space being altered.

FIG. 15 illustrates an example of changing contextual settings of acomputing device in response to a change in context.

FIG. 16 illustrates an example computing system in which techniques ofradar-enabled sensor fusion may be implemented.

DETAILED DESCRIPTION Overview

Conventional sensor techniques are often limited and inaccurate due toinherent weaknesses associated with a given type of sensor. For example,motion can be sensed through data provided by an accelerometer, yet theaccelerometer data may not be useful to determine a source of themotion. In other cases, a proximity sensor may provide data sufficientto detect proximity with an object, but an identity of the object maynot be determinable from the proximity data. As such, conventionalsensors have weaknesses or blind spots that can result in inaccurate orincomplete sensing of a device's surrounding, including the device'srelation to a user.

Apparatuses and techniques are described herein that implementradar-enabled sensor fusion. In some embodiments, respective strengthsof sensors are combined with a radar to mitigate a respective weaknessof each sensor. For example, a surface radar feature of a user's facecan be combined with imagery of a red-green-blue (RGB) camera to improveaccuracy of facial recognition application. In other cases, a radarmotion feature, which is able to track fast motion, is combined withimagery of an RGB sensor, which excels at capturing spatial information,to provide an application that is capable of detecting fast spatialmovements.

In yet other cases, radar surface features can be augmented withorientation or directional information from an accelerometer to enablemapping of a device's environment (e.g., rooms or spaces). In suchcases, the device may learn or detect contexts in which the device isoperating thereby enabling various contextual features and settings ofthe device. These are but a few examples of ways in which radar can beleveraged for sensor fusion or contextual sensing, which are describedherein. The following discussion first describes an operatingenvironment, followed by techniques that may be employed in thisenvironment, and ends with example systems.

Operating Environment

FIG. 1 illustrates a computing device through which radar-enabled sensorfusion can be enabled. Computing device 102 is illustrated with variousnon-limiting example devices, smart-glasses 102-1, a smart-watch 102-2,a smartphone 102-3, a tablet 102-4, a laptop computer 102-5, and agaming system 102-6, though other devices may also be used, such as homeautomation and control systems, entertainment systems, audio systems,other home appliances, security systems, netbooks, automobiles,smart-appliances, and e-readers. Note that the computing device 102 canbe wearable, non-wearable but mobile, or relatively immobile (e.g.,desktops and appliances).

The computing device 102 includes one or more computer processors 104and computer-readable media 106, which includes memory media and storagemedia. Applications and/or an operating system (not shown) embodied ascomputer-readable instructions on computer-readable media 106 can beexecuted by processors 104 to provide some of the functionalitiesdescribed herein. The computer-readable media 106 also includessensor-based applications 108, a sensor fusion engine 110, and a contextmanager 112, which are described below.

The computing device 102 may also include one or more network interfaces114 for communicating data over wired, wireless, or optical networks anda display 116. The network interface 114 may communicate data over alocal-area-network (LAN), a wireless local-area-network (WLAN), apersonal-area-network (PAN), a wide-area-network (WAN), an intranet, theInternet, a peer-to-peer network, point-to-point network, a meshnetwork, and the like. The display 116 can be integral with thecomputing device 102 or associated with it, such as with the gamingsystem 102-6.

The computing device 102 includes one or more sensors 118, which enablethe computing device 102 to sense various properties, variances,stimuli, or characteristics of an environment in which computing device102 operates. For example, the sensors 118 may include various motionsensors, light sensors, acoustic sensors, and magnetic sensors.Alternately or additionally, sensors 118 enable interaction with, orreceive input from, a user of computing device 102. The use andimplementation of the sensors 118 varies and is described below.

The computing device 102 may also be associated with or include a radarsensor 120. The radar sensor 120 represents functionality thatwirelessly detects targets through the transmission and reception ofradio frequency (RF) or radar signals. The radar sensor 120 can beimplemented as a system and/or radar-enabled component embedded withinthe computing device 102, such as a System-on-Chip (SoC) orsensor-on-chip. It is to be appreciated, however, that the radar sensor120 can be implemented in any other suitable manner, such as one or moreIntegrated Circuits (ICs), as a processor with embedded processorinstructions or configured to access a memory having processorinstructions stored thereon, as hardware with embedded firmware, aprinted circuit board assembly with various hardware components, or anycombination thereof Here, the radar sensor 120 includes radar-emittingelement 122, antenna(s) 124, and digital signal processor 126, which canbe used in concert to wirelessly detect various types of targets in theenvironment of the computing device 102.

Generally, radar-emitting element 122 is configured to provide a radarfield. In some cases, the radar field is configured to at leastpartially reflect off one or more target objects. In some cases, thetarget objects include device users or other people present in theenvironment of the computing device 102. In other cases, the targetobjects include physical features of the user, such as hand motion,breathing rates, or other physiological features. The radar field canalso be configured to penetrate fabric or other obstructions and reflectfrom human tissue. These fabrics or obstructions can include wood,glass, plastic, cotton, wool, nylon and similar fibers, and so forth,while reflecting from human tissues, such as a person's hand

A radar field provided by the radar-emitting element 122 can be a smallsize, such as zero or one millimeters to 1.5 meters, or an intermediatesize, such as one to 30 meters. It is to be appreciated that these sizesare merely for discussion purposes, and that any other suitable size orrange of radar field can be used. For example, when the radar field hasan intermediate size, the radar sensor 120 can be configured to receiveand process reflections of the radar field to provide large-bodygestures based on reflections from human tissue caused by body, arm, orleg movements.

In some aspects, the radar field can be configured to enable the radarsensor 120 to detect smaller and more-precise gestures, such asmicro-gestures. Example intermediate-sized radar fields include those inwhich a user makes gestures to control a television from a couch, changea song or volume from a stereo across a room, turn off an oven or oventimer (a near field would also be useful here), turn lights on or off ina room, and so forth. The radar sensor 120, or emitter thereof, can beconfigured to emit continuously modulated radiation, ultra-widebandradiation, or sub-millimeter-frequency radiation.

The antenna(s) 124 transmit and receive RF signals of the radar sensor120. In some cases, the radar-emitting element 122 is coupled with theantennas 124 to transmit a radar field. As one skilled in the art willappreciate, this is achieved by converting electrical signals intoelectromagnetic waves for transmission, and vice versa for reception.The radar sensor 120 can include one or an array of any suitable numberof antennas in any suitable configuration. For instance, any of theantennas 124 can be configured as a dipole antenna, a parabolic antenna,a helical antenna, a planar antenna, an inverted-F antenna, a monopoleantenna, and so forth. In some embodiments, the antennas 124 areconstructed or formed on-chip (e.g., as part of a SoC), while in otherembodiments, the antennas 124 are separate components, metal,dielectrics, hardware, etc. that attach to, or are included within,radar sensor 120.

A first antenna 124 can be single-purpose (e.g., a first antenna can bedirected towards transmitting signals, and a second antenna 124 can bedirected towards receiving signals), or multi-purpose (e.g., an antennais directed towards transmitting and receiving signals). Thus, someembodiments utilized varying combinations of antennas, such as anembodiment that utilizes two single-purpose antennas configured fortransmission in combination with four single-purpose antennas configuredfor reception. The placement, size, and/or shape of the antennas 124 canbe chosen to enhance a specific transmission pattern or diversityscheme, such as a pattern or scheme designed to capture informationabout the environment, as further described herein.

In some cases, the antennas 124 can be physically separated from oneanother by a distance that allows the radar sensor 120 to collectivelytransmit and receive signals directed to a target object over differentchannels, different radio frequencies, and different distances. In somecases, the antennas 124 are spatially distributed to supporttriangulation techniques, while in others the antennas are collocated tosupport beamforming techniques. While not illustrated, each antenna cancorrespond to a respective transceiver path that physically routes andmanages the outgoing signals for transmission and the incoming signalsfor capture and analysis.

The digital signal processor 126 ((DSP) or digital signal processingcomponent) generally represents operations related to digitallycapturing and processing a signal. For instance, the digital signalprocessor 126 samples analog RF signals received by the antenna(s) 124to generate radar data (e.g., digital samples) that represents the RFsignals, and then processes this radar data to extract information aboutthe target object. In some cases, the digital signal processor 126performs a transform on the radar data to provide a radar feature thatdescribes target characteristics, position, or dynamics. Alternately oradditionally, the digital signal processor 126 controls theconfiguration of signals generated and transmitted by the radar-emittingelement 122 and/or antennas 124, such as configuring a plurality ofsignals to form a specific diversity or beamforming scheme.

In some cases, the digital signal processor 126 receives inputconfiguration parameters that control an RF signal's transmissionparameters (e.g., frequency channel, power level, etc.), such as throughthe sensor-based applications 108, sensor fusion engine 110, or contextmanager 112. In turn, the digital signal processor 126 modifies the RFsignal based upon the input configuration parameter. At times, thesignal processing functions of the digital signal processor 126 areincluded in a library of signal processing functions or algorithms thatare also accessible and/or configurable via the sensor-basedapplications 108 or application programming interfaces (APIs). Thedigital signal processor 126 can be implemented in hardware, software,firmware, or any combination thereof.

FIG. 2 illustrates example types and configurations of the sensors 118that can be used to implement embodiments of radar-enabled sensor fusiongenerally at 200. These sensors 118 enable the computing device 102 tosense various properties, variances, stimuli, or characteristics of anenvironment in which computing device 102 operates. Data provided by thesensors 118 is accessible to other entities of the computing device,such as the sensor fusion engine 110 or the context manager 112.Although not shown, the sensors 118 may also include global-positioningmodules, micro-electromechanical systems (MEMS), resistive touchsensors, and so on. Alternately or additionally, the sensors 118 canenable interaction with, or receive input from, a user of the computingdevice 102. In such a case, the sensors 118 may include piezoelectricsensors, touch sensors, or input sensing-logic associated with hardwareswitches (e.g., keyboards, snap-domes, or dial-pads), and so on.

In this particular example, the sensors 118 include an accelerometer 202and gyroscope 204. These and other motion and positional sensors, suchas motion sensitive MEMS or global positioning systems (GPSs) (notshown), are configured to sense movement or orientation of the computingdevice 102. The accelerometer 202 or gyroscope 204 can sense movement ororientation of the device in any suitable aspect, such as inone-dimension, two-dimensions, three-dimensions, multi-axis, combinedmulti-axis, and the like. Alternately or additionally, positionalsensor, such as a GPS, may indicate a distance traveled, rate of travel,or an absolute or relative position of the computing device 102. In someembodiments, the accelerometer 202 or gyroscope 204 enable computingdevice 102 to sense gesture inputs (e.g., a series of position and/ororientation changes) made when a user moves the computing device 102 ina particular way.

The computing device 102 also includes a hall effect sensor 206 andmagnetometer 208. Although not shown, the computing device 102 may alsoinclude a magneto-diode, magneto-transistor, magnetic sensitive MEMS,and the like. These magnetic field-based sensors are configured to sensemagnetic field characteristics around computing device 102. For example,the magnetometer 208 may sense a change in magnetic field strength,magnetic field direction, or magnetic field orientation. In someembodiments, computing device 102 determines proximity with a user oranother device based on input received from the magnetic field-basedsensors.

A temperature sensor 210 of the computing device 102 can sense atemperature of a housing of the device or ambient temperature of thedevice's environment. Although not shown, the temperature sensor 210 mayalso be implemented in conjunction with a humidity sensor that enablesmoisture levels to be determined. In some cases, the temperature sensorcan sense a temperature of a user that is holding, wearing, or carrying,the computing device 102. Alternately or additionally, the computingdevice may include an infrared thermal sensor that can sense temperatureremotely or without having physical contact with an object of interest.

The computing device 102 also includes one or more acoustic sensors 212.The acoustic sensors can be implemented as microphones or acoustic wavesensors configured to monitor sound of an environment that computingdevice 102 operates. The acoustic sensors 212 are capable of receivingvoice input of a user, which can then be processed by a DSP or processorof computing device 102. Sound captured by the acoustic sensors 212 maybe analyzed or measured for any suitable component, such as pitch,timbre, harmonics, loudness, rhythm, envelope characteristics (e.g.,attack, sustain, decay), and so on. In some embodiments, the computingdevice 102 identifies or differentiates a user based on data receivedfrom the acoustic sensors 212.

Capacitive sensors 214 enable the computing device 102 to sense changesin capacitance. In some cases, the capacitance sensors 214 areconfigured as touch sensors that can receive touch input or determineproximity with a user. In other cases, the capacitance sensors 214 areconfigured to sense properties of materials proximate a housing of thecomputing device 102. For example, the capacitance sensors 214 mayprovide data indicative of the devices proximity with respect to asurface (e.g., table or desk), body of a user, or the user's clothing(e.g., clothing pocket or sleeve). Alternately or additionally, thecapacitive sensors may be configured as a touch screen or other inputsensor of the computing device 102 through which touch input isreceived.

The computing device 102 may also include proximity sensors 216 thatsense proximity with objects. The proximity sensors may be implementedwith any suitable type of sensor, such as capacitive or infrared (IR)sensors. In some cases, the proximity sensor is configured as ashort-range IR emitter and receiver. In such cases, the proximity sensormay be located within a housing or screen of the computing device 102 todetect proximity with a user's face or hand. For example, a proximitysensor 216 of a smart-phone may enable detection of a user's face, suchas during a voice call, in order to disable a touch screen of thesmart-phone to prevent the reception of inadvertent user input.

An ambient light sensor 218 of the computing device 102 may include aphoto-diode or other optical sensors configured to sense an intensity,quality, or changes in light of the environment. The light sensors arecapable of sensing ambient light or directed light, which can then beprocessed by the computing device 102 (e.g., via a DSP) to determineaspects of the device's environment. For example, changes in ambientlight may indicate that a user has picked up the computing device 102 orremoved the computing device 102 from his or her pocket.

In this example, the computing device also includes a red-green-bluesensor 220 (RGB sensor 220) and an infrared sensor 222. The RGB sensor220 may be implemented as a camera sensor configured to capture imageryin the form of images or video. In some cases, the RGB sensor 220 isassociated with a light-emitting diode (LED) flash increase luminosityof the imagery in low-light environments. In at least some embodiments,the RGB sensor 220 can be implemented to capture imagery associated witha user, such as a user's face or other physical features that enableidentification of the user.

The infrared sensor 222 is configured to capture data in the infraredfrequency spectrum, and may be configured to sense thermal variations oras an infrared (IR) camera. For example, the infrared sensor 222 may beconfigured to sense thermal data associated with a user or other peoplein the device's environment. Alternately or additionally, the infraredsensor may be associated with an IR LED and configured to senseproximity with or distance to an object.

In some embodiments, the computing device includes a depth sensor 224,which may be implemented in conjunction with the RGB senor 220 toprovide RGB-enhanced depth information. The depth sensor 222 may beimplemented as a single module or separate components, such as an IRemitter, IR camera, and depth processor. When implemented separately,the IR emitter emits IR light that is received by the IR camera, whichprovides IR imagery data to the depth processor. Based on knownvariables, such as the speed of light, the depth processor of the depthsensor 224 can resolve distance to a target (e.g., time-of-flightcamera). Alternately or additionally, the depth sensor 224 may resolve athree-dimensional depth map of the object's surface or environment ofthe computing device.

From a power consumption viewpoint, each of the sensors 118 may consumea different respective amount of power while operating. For example, themagnetometer 208 or acoustic sensor 212 may consume tens of milliamps tooperate while the RGB sensor, infrared sensor 222, or depth sensor 224may consume hundreds of milliamps to operate. In some embodiments, powerconsumption of one or more of the sensors 118 is known or predefinedsuch that lower power sensors can be activated in lieu of other sensorsto obtain particular types of data while conserving power. In manycases, the radar sensor 120 can operate, either continuously orintermittently, to obtain various data while consuming less power thanthe sensors 118. In such cases, the radar sensor 120 may operate whileall or most of the sensors 118 are powered-down to conserve power of thecomputing device 102. Alternately or additionally, a determination canbe made, based on data provided by the radar sensor 120, to activate oneof the sensors 118 to obtain additional sensor data.

FIG. 3 illustrates example configurations of the radar sensor 120 andradar fields provided thereby generally at 300. In the context of FIG.3, two example configurations of the radar sensor 120 are illustrated, afirst in which a radar sensor 302-1 is embedded in a gaming system 304and a second in which a radar sensor 302-2 is embedded in a television306. The radar sensors 302-1 and 302-2 may be implemented similarly toor differently from each other or the radar sensors described elsewhereherein. In the first example, the radar sensor 302-1 provides a nearradar field to interact with the gaming system 304, and in the second,the radar sensor 302-2 provides an intermediate radar field (e.g., aroom size) to interact with the television 306. These radar sensors302-1 and 302-2 provide near radar field 308-1 and intermediate radarfield 308-2, respectively, and are described below.

The gaming system 304 includes, or is associated with, the radar sensor302-1. These devices work together to improve user interaction with thegaming system 304. Assume, for example, that the gaming system 304includes a touch screen 310 through which content display and userinteraction can be performed. This touch screen 310 can present somechallenges to users, such as needing a person to sit in a particularorientation, such as upright and forward, to be able to touch thescreen. Further, the size for selecting controls through touch screen310 can make interaction difficult and time-consuming for some users.Consider, however, the radar sensor 302-1, which provides near radarfield 308-1 enabling a user's hands to interact with desktop computer304, such as with small or large, simple or complex gestures, includingthose with one or two hands, and in three dimensions. As is readilyapparent, a large volume through which a user may make selections can besubstantially easier and provide a better experience over a flatsurface, such as that of touch screen 310.

Similarly, consider the radar sensor 302-2, which provides theintermediate radar field 308-2. Providing a radar-field enables avariety of interactions with a user positioned in front of thetelevision. For example, the user may interact with the television 306from a distance and through various gestures, ranging from handgestures, to arm gestures, to full-body gestures. By so doing, userselections can be made simpler and easier than a flat surface (e.g.,touch screen 310), a remote control (e.g., a gaming or televisionremote), and other conventional control mechanisms. Alternately oradditionally, the television 306 may determine, via the radar sensor302-2, an identity of the user, which can be provided to sensor-basedapplications to implement other functions (e.g., content control).

FIG. 4 illustrates another example configuration of the radar sensor anda penetrating radar field provided thereby at 400. In this particularexample, a surface to which the radar field is applied human tissue. Asshown, a hand 402 having a surface radar field 404 provided by the radarsensor 120 (of FIG. 1) that is included in a laptop 406. Aradar-emitting element 122 (not shown) provides the surface radar field404 that penetrates a chair 408 and is applied to the hand 402. In thiscase, the antennas 124 are configured to receive a reflection caused byan interaction on the surface of the hand 402 that penetrates (e.g.,reflects back through) the chair 408. Alternately, the radar sensor 120can be configured to provide and receive reflections through fabric,such as when a smart-phone is placed in a user's pocket. Thus, the radarsensor 120 may map or scan spaces through an optical occlusion, such asfabric, clothing, and other non-transparent material.

In some embodiments, the digital signal processor 126 is configured toprocess the received reflection signal from the surface sufficient toprovide radar data usable to identify the hand 402 and/or determine agesture made thereby. Note that with the surface radar field 404,another hand may by identified or interact to perform gestures, such asto tap on the surface on the hand 402, thereby interacting with thesurface radar field 404. Example gestures include single andmulti-finger swipe, spread, squeeze, non-linear movements, and so forth.Or the hand 402 may simply move or change shape to cause reflections,thereby also performing an occluded gesture.

With respect to human-tissue reflection, reflecting radar fields canprocess these fields to determine identifying indicia based on thehuman-tissue reflection, and confirm that the identifying indiciamatches recorded identifying indicia for a person, such asauthentication for a person permitted to control a correspondingcomputing device. These identifying indicia can include variousbiometric identifiers, such as a size, shape, ratio of sizes, cartilagestructure, and bone structure for the person or a portion of the person,such as the person's hand These identify indicia may also be associatedwith a device worn by the person permitted to control the mobilecomputing device, such as device having a unique or difficult-to-copyreflection (e.g., a wedding ring of 14 carat gold and three diamonds,which reflects radar in a particular manner).

In addition, the radar sensor systems can be configured so thatpersonally identifiable information is removed. For example, a user'sidentity may be treated so that no personally identifiable informationcan be determined for the user, or a user's geographic location may begeneralized where location information is obtained (such as to a city,ZIP code, or state level), so that a particular location of a usercannot be determined. Thus, the user may have control over whatinformation is collected about the user, how that information is used,and what information is provided to the user.

FIG. 5 illustrates an example configuration of components capable ofimplementing radar-enabled sensor fusion generally at 500, including thesensor fusion engine 110 and context manager 112. Although shown asseparate entities, the radar sensor 120, sensor fusion engine 110,context manager 112, and other entities may be combined with oneanother, organized differently, or communicate directly or indirectlythrough interconnections or data buses not shown. Accordingly, theimplementation of the sensor fusion engine 110 and context manager 112shown in FIG. 5 is intended to provide a non-limiting example of ways inwhich these entities and others described herein may interact toimplement radar-enabled sensor fusion.

In this example, the sensor fusion engine includes a radar signaltransformer 502 (signal transformer 502) and a radar feature extractor504 (feature extractor 504). Although shown as separate entitiesembodied on the sensor fusion engine 110, the signal transformer 502 andfeature abstractor 504 may also be implemented by, or within, thedigital signal processor 126 of the radar sensor 120. The sensor fusionengine 110 is communicably coupled with the sensors 118, from whichsensor data 506 is received. The sensor data 506 may include anysuitable type of raw or pre-processed sensor data, such as datacorresponding to any type of the sensors described herein. The sensorfusion engine 110 is also operably coupled with the radar sensor 120,which provides radar data 508 to the sensor fusion engine 110.Alternately or additionally, the radar data 508 provided by the radarsensor 120 may comprise real-time radar data, such as raw datarepresenting reflection signals of a radar field as they are received bythe radar sensor 120.

In some embodiments, the signal transformer 502 transforms the raw radardata representing reflection signals into radar data representations. Insome cases, this includes performing signal pre-processing on the rawradar data. For example, as an antenna receives reflected signals, someembodiments sample the signals to generate a digital representation ofthe raw incoming signals. Once the raw data is generated, the signaltransformer 502 pre-processes the raw data to clean up the signals orgenerate versions of the signals in a desired frequency band or desireddata format. Alternately or additionally, pre-processing the raw datamay include filtering the raw data to reduce a noise floor or removealiasing, resampling the data to obtain to a different sample rate,generating a complex representation of the signals, and so on. Thesignal transformer 502 can pre-process the raw data based on defaultparameters, while in other cases the type and parameters of thepre-processing is configurable, such as by the sensor fusion engine 110or the context manager 112.

The signal transformer 502 transforms the received signal data into oneor more different data representations or data transforms. In somecases, the signal transformer 502 combines data from multiple paths andcorresponding antennas. The combined data may include data from variouscombinations of transmit paths, receive paths, or combined transceiverpaths of the radar sensor 120. Any suitable type of data fusiontechnique can be used, such as weighted integration to optimize aheuristic (e.g., signal-to-noise (SNR) ratio or minimum mean squareerror (MMSE)), beamforming, triangulation, and the like.

The signal transformer 502 may also generate multiple combinations ofsignal data for different types of feature extraction, and/or transformsthe signal data into another representation as a precursor to featureextraction. For example, the signal transformer 502 can process thecombined signal data to generate a three-dimensional (3D) spatialprofile of the target object. However, any suitable type of algorithm ortransform can be used to generate a view, abstraction, or version of theraw data, such as an I/Q transformation that yields a complex vectorcontaining phase and amplitude information related to the target object,a beamforming transformation that yields a spatial representation oftarget objects within range of a gesture sensor device, or arange-Doppler algorithm that yields target velocity and direction. Othertypes of algorithms and transforms may include a range profile algorithmthat yields target recognition information, a micro-Doppler algorithmthat yields high-resolution target recognition information, and aspectrogram algorithm that yields a visual representation of thecorresponding frequencies, and so forth.

As described herein, the raw radar data can be processed in several waysto generate respective transformations or combined signal data. In somecases, the same raw data can be analyzed or transformed in multipleways. For instance, a same capture of raw data can be processed togenerate a 3D profile, target velocity information, and targetdirectional movement information. In addition to generatingtransformations of the raw data, the radar signal transformer canperform basic classification of the target object, such as identifyinginformation about its presence, a shape, a size, an orientation, avelocity over time, and so forth. For example, some embodiments use thesignal transformer 502 to identify a basic orientation of a hand bymeasuring an amount of reflected energy off of the hand over time.

These transformations and basic classifications can be performed inhardware, software, firmware, or any suitable combination. At times, thetransformations and basic classifications are performed by digitalsignal processor 126 and/or the sensor fusion engine 110. In some cases,the signal transformer 502 transforms the raw radar data or performs abasic classification based upon default parameters, while in other casesthe transformations or classifications are configurable, such as throughthe sensor fusion engine 110 or context manager 112.

The feature abstractor 504 receives the transformed representations ofthe radar data from the signal transformer 502. From these datatransforms, the feature abstractor 504 resolves, extracts, or identifiesone or more radar features 510. These radar features 510 may indicatevarious properties, dynamics, or characteristics of a target and, inthis example, include detection features 512, reflection features 514,motion features 516, position features 518, and shape features 520.These features are described by way of example only, and are notintended to limit ways in which the sensor fusion engine extractsfeature or gesture information from raw or transformed radar data. Forexample, the radar feature extractor 504 may extract alternate radarfeatures, such as range features or image features, from radar datarepresentations provided by the signal transformer 502.

The detection features 512 may enable the sensor fusion engine 110 todetect a presence of a user, other people, or objects in an environmentof the computing device 102. In some cases, a detection featureindicates a number of targets in a radar field or a number of targets ina room or space that is swept by a radar field. The reflection features514 may indicate a profile of energy reflected by the target, such asreflected energy over time. This can be effective to enable velocity ofa target's motion to be tracked over time. Alternately or additionally,the reflection feature may indicate an energy of a strongest componentor a total energy of a moving target.

The motion features 516 may enable the fusion engine 110 to trackmovement or motion of a target in or through a radar field. In somecases, the motion feature 516 includes a velocity centroid in one orthree dimensions, or a phase-based fine target displacement in onedimension. Alternately or additionally, the motion feature may include atarget velocity or a 1D velocity dispersion. In some embodiments, theposition features 518 include spatial 2D or 3D coordinates of a targetobject. The position features 518 may also be useful to range ordetermine distance to a target object.

The shape features 520 indicate a shape of a target or surface, and mayinclude a spatial dispersion. In some cases, the sensor fusion engine110 can scan or beam form different radar fields to build a 3Drepresentation of a target or environment of the computing device 102.For example, the shape features 520 and other of the radar features 510can be combined by the sensor fusion engine 110 to construct a uniqueidentifier (e.g., a fingerprint) of a particular room or space.

In some embodiments, the feature abstractor 504 builds on a basicclassification identified by the signal transformer 502 for featureextraction or abstraction. Consider the above example in which thesignal transformer 502 classifies a target object as a hand The featureabstractor 504 can build from this basic classification to extract lowerresolution features of the hand In other words, if the featureabstractor 504 is provided information identifying the target object asa hand, then the feature abstractor 504 uses this information to lookfor hand-related features (e.g., finger tapping, shape gestures, orswipe movements) instead of head-related features, (e.g., an eye blink,mouthing a word, or head-shaking movement).

As another example, consider a scenario in which the signal transformer502 transforms the raw radar data into a measure of the target object'svelocity over time. In turn, the feature abstractor 504 uses thisinformation to identify a finger fast-tap motion by using a thresholdvalue to compare the target object's velocity of acceleration to thethreshold value, a slow-tap feature, and so forth. Any suitable type ofalgorithm can be used to extract a feature, such as machine-learningalgorithms implemented by a machine-learning component (not shown) ofthe digital signal processor 126.

In various embodiments, the sensor fusion engine 110 combines oraugments the radar features 510 with sensor data 506 from the sensors118. For example, the sensor fusion engine 110 may apply a singlealgorithm to extract, identify, or classify a feature, or apply multiplealgorithms to extract a single feature or multiple features. Thus,different algorithms can be applied to extract different types offeatures on a same set of data, or different sets of data. Based on theradar feature, the sensor fusion engine 110 can activate a particularsensor to provide data that is complimentary or supplemental to theradar feature. By so doing, an accuracy or validity of the radarfeatures can be improved with the sensor data.

The sensor fusion engine 110 provides or exposes the sensor data 506,radar data 508, or various combinations thereof to sensor-basedapplications 108 and the context manager 112. For example, the sensorfusion engine 110 may provide, to the sensor-based applications 108,radar data augmented with sensor data or radar data that is verifiedbased on sensor data. The sensor-based applications 108 may include anysuitable application, function, utility, or algorithm that leveragesinformation or knowledge about the computing device 102's environment orrelation thereto in order to provide device functionalities or to alterdevice operations.

In this particular example, the sensor-based applications 108 includeproximity detection 522, user detection 524, and activity detection 526.The proximity detection application 522 may detect, based on sensor dataor radar data, proximity with a user or other objects. For example, theproximity detection application 522 may use detection radar features 512to detect an approaching object and then switch to proximity sensor datato confirm proximity with a user. Alternately or additionally, theapplication may leverage a shape radar feature 520 to verify that theapproaching object is a user's face, and not another similar sized largebody object.

The user detection application 524 may detect, based on sensor data orradar data, presence of a user. In some cases, the user detectionapplication 524 also tracks the user when the user is detected in theenvironment. For example, the user detection application 524 may detectthe presence based on a detection radar feature 512 and shape radarfeature 520 that matches a known 3D profile of the user. The userdetection application 524 can also verify detection of the user throughimage data provided by the RGB sensor 220 or voice data provided by theacoustic sensors 212.

In some embodiments, the activity detection application 526 uses thesensor data and the radar data to detect activity in the computingdevice 102′s environment. The activity detection application 526 maymonitor radar for detection features 512 and motion features 516.Alternately or additionally, the activity detection application 526 canuse the acoustic sensors 212 to detect noise, and the RGB sensor 220 ordepth sensor 224 to monitor movement.

The sensor-based applications also include biometric recognition 528,physiologic monitor 530, and motion identification 532. The biometricrecognition application 528 may use sensor data and radar data tocapture or obtain biometric characteristics of a user that are useful toidentify that user, such as to implement facial recognition. Forexample, the biometric recognition application 528 can use a shape radarfeature 520 to obtain a 3D map of skeletal structure of the user's faceand a color image from the RGB sensor 220 to confirm the user'sidentity. Thus, even if an imposter was able to forge an appearance ofthe user, the imposter would not be able to replicate the exact facialstructure of the user and thus fail identification through the biometricrecognition application 528.

The physiological monitor application 530 can detect or monitor medicalaspects of a user, such as breathing, heart rate, reflexes, fine motorskills, and the like. To do so, the physiological monitor application530 may use radar data, such as to track motion of user's chest, monitorarterial flow, subdermal muscle contraction, and the like. Thephysiological monitor application 530 can monitor other sensors of thedevice for supplemental data, such as acoustic, thermal (e.g.,temperature), image (e.g., skin or eye color), and motion (e.g.,tremors) data. For example, the physiological monitor application 530may monitor a user's breathing patterns with radar motion features 516,breath noises recorded by the acoustic sensors 212, and heat signaturesof exhaled air captured by the infrared sensor 222.

The motion identification application 532 can use the radar data andsensor data to identify various motion signatures. In some cases, themotion radar features 516 or other radar features are useful for trackmotion. In such cases, the motions may be too fast for accurate captureby the RGB sensor 220 or the depth sensor 224. By using the radarfeatures, which are able to track very fast motion, the motionidentification application 532 can track motion and leverage image datafrom the RGB sensor 220 to provide additional spatial information. Thus,the sensor fusion engine 110 and the motion identification application532 are able to track a fast moving object with corresponding spatialinformation.

The gesture detection application 534 of the sensor-based applications108 performs gesture recognition and mapping. For instance, consider acase where a finger tap motion feature has been extracted. The gesturedetection application 534 can use this information, sound data from theacoustic sensors 212, or image data from the RGB sensor 220 to identifythe feature as a double-click gesture. The gesture detection application534 may use a probabilistic determination of which gesture has mostlikely occurred based upon the radar data and sensor data provided bythe sensor fusion engine 110, and how this information relates to one ormore previously learned characteristics or features of various gestures.For example, a machine-learning algorithm can be used to determine howto weight various received characteristics to determine a likelihoodthese characteristics correspond to particular gestures (or componentsof the gestures).

The context manager 112 may access the sensor-based applications 108,sensor data 506, or the radar features 510 of the sensor fusion engine110 to implement radar-based contextual sensing. In some embodiments,the radar data 508 can be combined with sensor data 506 to provide mapsof spaces or rooms in which the computing device 102 operates. Forexample, position and inertial sensor data can be leveraged to implementsynthetic aperture techniques for capturing and meshing 3D radarimagery. Thus, as the device moves through an environment, the contextmanager 112 can construct detailed or high resolution 3D maps of variousspaces and rooms. Alternately or additionally, the 3D imagery can becaptured through optical occlusions or be used in combination with othertechniques of sensor fusion to improve activity recognition.

In this particular example, the context manager 112 includes contextmodels 536, device contexts 538, and context settings 540. The contextmodels 536 include physical models of various spaces, such asdimensions, geometry, or features of a particular room. In other words,a context model can be considered to describe the unique character ofparticular space, like a 3D fingerprint. In some cases, building thecontext models 536 is implemented via machine learning techniques andmay be performed passively as a device enters or passes through aparticular space. Device contexts 538 include and may describe multiplecontexts in which the computing device 102 may operate. These contextsmay include a standard set of work contexts, such as “meeting,” “do notdisturb,” “available,” “secure,” “private,” and the like. For example,the “meeting” context may be associated with the device being in aconference room, with multiple other coworkers and customers.Alternately or additionally, the device contexts 538 may be userprogrammable or custom, such as contexts for different rooms of a house,with each context indicating a respective level of privacy or securityassociated with that context.

Context settings 540 include various device or system settings that areconfigurable based on context or other environmental properties. Thecontext settings 540 may include any suitable type of device setting,such as ringer volume, ringer mode, display modes, connectivity tonetworks or other devices, and the like. Alternately or additionally,the context settings 540 may include any suitable type of systemsettings, such as security settings, privacy settings, network or deviceconnection settings, remote control features, and the like. For example,if a user walks into her home theater, the context manager 112 mayrecognize this context (e.g., “home theater”) and configure contextsettings by muting a device's alerts and configuring a wirelessinterface of the device to control audio/video equipment in the hometheater. This is just but one example of how context manager 112 candetermine and configure a device based on context.

Having described respective examples of a computing device 102, a sensorfusion engine 110, and a context manager 112 in accordance with one ormore embodiments, now consider a discussion of techniques that can beperformed by those and other entities described herein to implementradar-enabled sensor fusion.

Example Methods

FIGS. 6, 8, 10, 11, 13, and 14 depict methods for implementingradar-enabled sensor fusion and/or radar-based contextual sensing. Thesemethod is shown as sets of blocks that specify operations performed butare not necessarily limited to the order or combinations shown forperforming the operations by the respective blocks. For example,operations of different methods may be combined, in any order, toimplement alternate methods without departing from the conceptsdescribed herein. In portions of the following discussion, thetechniques may be described in reference may be made to FIGS. 1-5,reference to which is made for example only. The techniques are notlimited to performance by one entity or multiple entities operating onone device, or those described in these figures.

FIG. 6 depicts an example method 600 for augmenting radar data withsupplemental sensor data, including operations performed by the radarsensor 120, sensor fusion engine 110, or context manager 112.

At 602, a radar field is provided, such as one of the radar fields shownin FIG. 2 or 3. The radar field can be provided by a radar system orradar sensor, which may be implemented similar to or differently fromthe radar sensor 120 and radar-emitting element 122 of FIG. 1. The radarfield provided may comprise a broad beam, full contiguous radar field ora directed narrow beam, scanned radar field. In some cases, the radarfield is provided at a frequency approximate a 60 GHz band, such as57-64 GHz or 59-61 GHz, though other frequency bands can be used.

By way of example, consider FIG. 7 in which a laptop computer 102-5includes a radar sensor 120 at 700 and is capable of implementingradar-enabled sensor fusion. Here, assume that a user 702 is using agesture driven control menu of the laptop 102-5 to play a first personshooter (FPS) video game. The radar sensor 120 provides radar field 704to capture movement of the user 702 to enable game control.

At 604, one or more reflection signals are received that correspond to atarget in the radar field. The radar reflection signals may be receivedas a superposition of multiple points of a target object in the radarfield, such as a person or object within or passing through the radarfield. In the context of the present example, reflection signals fromthe user's hand are received by the radar sensor 120.

At 606, the one or more reflection signals are transformed into radardata representations. The reflection signals may be transformed usingany suitable signal processing, such as by performing a range-Dopplertransform, range profile transform, micro-Doppler transform, I/Qtransform, or spectrogram transform. Continuing the ongoing example, theradar sensor performs a range-Doppler transform to provide targetvelocity and direction information for the user's hand.

At 608, a radar feature indicative of a characteristic of the target isextracted from the radar data. The radar feature may provide a real-timemeasurement of the characteristic of the target, position of the target,or dynamics of the target. The radar feature may include any suitabletype of feature, such as a detection feature, reflection feature, motionfeature, position feature, or shape feature, examples of which aredescribed herein. In the context of the present example, reflectionradar features and motion radar features of the user's hand areextracted from the radar data.

At 610, a sensor is activated based on the radar feature. The sensor canbe activated to provide supplemental data. In some cases, a sensor isselected for activation based on the radar feature or a type of theradar feature. For example, an RGB or infrared sensor can be activatedto provide supplemental sensor data for a surface feature or motionfeature. In other cases, an accelerometer or gyroscope can be activatedto obtain supplemental data for a motion feature or position feature. Inyet other cases, data may be received from a microphone or depth sensorto improve a detection feature. Continuing the ongoing example, thesensor fusion engine 110 activates the RGB sensor 220 of the laptop102-5 to capture spatial information.

At 612, the radar feature is augmented with the supplemental sensordata. This may include combining, or fusing, the radar feature andsensor data to provide a more-accurate or more-precise radar feature.Augmenting the radar feature may include improving an accuracy orresolution of the radar feature based on supplemental or complimentarysensor data. Examples of this sensor fusion may include using sensordata to increase a positional accuracy of the radar feature, mitigate afalse detection attributed to the radar feature, increase a spatialresolution of the radar feature, increase a surface resolution of theradar feature, or improve a classification precision of the radarfeature. In the context of the present example, the sensor fusion engine110 combines the motion radar features 516 and RGB information toprovide sensor information that spatially captures a very fast movement.In some cases, the RGB sensor 220 would not be able to detect or capturesuch motion due to inherent limitations of the sensor.

At 614, the augmented radar feature is provided to a sensor-basedapplication. This can be effective to increase performance of thesensor-based application. In some cases, the augmented radar featureincreases accuracy of detection applications, such as proximitydetection, user detection, activity detection, gesture detection, andthe like. In such cases, sensor data may be used to eliminate falsedetection, such as by confirming or disproving detection of the target.In other cases, the augmented radar feature may improve consistency ofthe application. Concluding the present example, the fused radar datafeatures are passed to the gesture detection application 534, whichpasses a gesture to the FPS video game as game control input.

FIG. 8 illustrates an example method for low-power sensor fusion,including operations performed by the radar sensor 120, sensor fusionengine 110, or context manager 112.

At 802, a radar sensor of a device is monitored for changes in reflectedsignals of a radar field. The radar sensor may provide a continuous orintermittent radar field from which the reflected signals are received.In some cases, the radar sensor is a lower-power sensor of a device,which consumes less power while operating than other sensors of thedevice. The changes in the reflected signals may be caused by movementof the device or movement of a target within the device's environment.By way of example, consider environment 900 of FIG. 9 in which aradar-enabled television 902 is being watched by a first user 904 in aliving room. Here, assume that user 904 begins reading a magazine andthat a second user 906 enters the living room. A radar sensor 120 of thetelevision detects changes in reflected radar signals caused by theseactions of the first and second users.

At 804, the reflected signals are transformed to detect a target in theradar field. In some cases, detection features are extracted from thetransformed radar data to confirm detection of the target in the radarfield. In other cases, shape features or motion features are extractedfrom the transformed radar data to identity physical characteristics thetarget or movement of the target in the radar field. In the context ofthe present example, detection radar features for the first and secondusers are extracted from the reflected radar signals.

At 806, responsive to detection of the target in the reflected signals,a higher-power sensor is activated from a low-power state to obtaintarget-related sensor data. For example, if a radar detection featureindicates movement or presence of a user in the radar field, an RGBsensor can be activated to capture imagery of the user. In other cases,a GPS module of the device may be activated in response to positionradar features or reflection radar features that indicate the device ismoving. Continuing the ongoing example, the sensor fusion engine 110activates the RGB sensor 220 of the television 902. The RGB sensor 220obtains face image data 908 of the first user 904 and face image data910 of the second user 906. This image data may be static images orvideo of the user's faces, such as to enable eye tracking or otherdynamic facial recognition features.

At 808, the target-related sensor data is passed to a sensor-basedapplication. The sensor-based application may include any suitableapplication, such as those described herein. In some cases, execution ofthe sensor-based application is initiated or resumed in response todetecting a particular activity or target in the radar field. Forexample, a surveillance application may be resumed in response tosensing activity features that indicate an unauthorized person enteringa controlled area. An RGB sensor can then pass image data to thesurveillance application. In the context of the present example in FIG.9, the RGB sensor passes the face image data 908 and face image data 910to a biometric recognition application 528.

Optionally at 810, radar features extracted from the transformed radardata are passed to the sensor-based application. In some cases, theradar features provide additional context for the sensor data passed tothe sensor-based application. For example, a position radar feature canbe passed to an application receiving RGB imagery to enable theapplication to tag targets in the imagery with respective locationinformation.

Continuing the ongoing example, the sensor fusion engine 110 passesrespective radar surface features of the user's faces to the biometricrecognition application 528. Here, the application may determine thatthe first user 904 is not watching the television (e.g., eye tracking),and determine that the second user 906 is interested in watching thetelevision 902. Using the face image data 910, the sensor fusion engine110 can identify the second user 906 and retrieve, based on hisidentity, a viewing history associated with this identity. The contextmanager 112 leverages this viewing history to change a channel of thetelevision to a last channel viewed by the second user 906.

At 812, the higher-power sensor is returned to a low-power state. Oncethe sensor data is passed to the sensor-based application, thehigher-power sensor can be returned to the low-power state to conservepower of the device. Because the radar sensor consumes relatively littlepower while providing an array of capabilities, the other sensors can beleft in low-power states until more sensor-specific data needs to beobtained. By so doing, power consumption of the device can be reduced,which is effective to increase run times for battery operated devices.From operation 812, the method 800 may return to operation 802 tomonitor the radar sensor for subsequent changes in the reflected signalsof the radar field. Concluding the present example, the RGB sensor 220is returned to a low-power state after changing the channel of thetelevision and may reside in the low-power state until further activityis detected by the radar sensor 120.

FIG. 10 illustrates an example method for enhancing sensor data withradar features, including operations performed by the radar sensor 120,sensor fusion engine 110, or context manager 112.

At 1002, a sensor of a device is monitored for environmental variances.The sensor may include any suitable type of sensor, such as thosedescribed in reference to FIG. 2 and elsewhere herein. The sensors maybe configured to monitor variances of a physical state of the device,such as device motion, or variance remote from the device, such asambient noise or light. In some cases, the sensor is monitored while thedevice is in a low-power state or by a low-power processor of the deviceto conserve device power.

At 1004, an environmental variance is detected via the sensor. Theenvironmental variance may include any suitable type of variance, suchas a user's voice, ambient noise, device motion, user proximity,temperature change, change in ambient light, and so on. The detectedenvironmental variance may be associated with a particular context,activity, user, and the like. For example, the environmental variancemay include a voice command from a user to wake the device from a sleepstate and unlock the device for use.

At 1006, responsive to detecting the environmental variance, a radarsensor is activated to provide a radar field. The radar sensor of thedevice may be activated to provide radar data that is supplemental orcomplimentary to data provided by the sensor. In some cases, the radarfield is configured based on a type of sensor that detects theenvironmental variance or sensor data that characterizes theenvironmental variance. For example, if user proximity is detected, theradar sensor is configured to provide a short-range radar field suitablefor identifying the user. In other cases, the radar sensor can beconfigured to provide a sweeping long-range radar field in response todetecting ambient noise or vibrations. In such cases, the long-rangeradar field can be used to detect activity or a location associated withthe source of the noise.

At 1008, reflection signals from the radar field are transformed toprovide radar data. The reflection signals may be transformed using anysuitable signal processing, such as by performing a range-Dopplertransform, range profile transform, micro-Doppler transform, I/Qtransform, or spectrogram transform. In some cases, a type of transformused to provide the radar data is selected based on a type of sensorthat detects the environmental variance or the data provided by thesensor.

At 1010, a radar feature is extracted from the radar data. The radarfeature can be extracted based on the environmental variance. In somecases, a type of radar feature is selected based on the environmentalvariance or a type of the environmental variance. For example, adetection feature or motion feature can be selected in response to anacoustic sensor detecting ambient noise. In other cases, the radarsensor can extract a position feature or shape feature in response to anaccelerometer sensing movement of the device.

At 1012, the sensor data is augmented with the radar feature to provideenhanced sensor data. This can be effective to increase an accuracy orconfidence rate associated with the sensor data. In other words, if thesensor has a weakness with respect to accuracy, range, or anothermeasurement, the radar data may compensate for this shortcomings andimprove the quality of the sensor data. For example, a surface featureof a user's face may confirm an identity of the user and a validity of avoice command received to unlock the device.

At 1014, the enhanced sensor data is exposed to a sensor-basedapplication. This can be effective to improve performance of thesensor-based application by improving an accuracy of the application,reducing an amount of sensor data used by the application, expandingfunctionality of the application, and the like. For example, amotion-based power state application that awakes the device in responseto movement may also authenticate a user and unlock the device based onenhanced sensor data that includes motion data and a surface feature ofthe user's facial structure.

FIG. 11 illustrates an example method for creating a 3D context modelfor a space of interest, including operations performed by the radarsensor 120, sensor fusion engine 110, or context manager 112.

At 1102, a radar sensor of a device is activated to obtain radar datafor a space or area of interest. The radar sensor may be activatedresponsive to movement of the device, such as inertial data or GPS dataindicating that the device is moving into the space or area. In somecases, the radar sensor is activated responsive to detecting unknowndevices, such as wireless access points, wireless appliances, or otherwireless devices in the space transmitting data detectable by a wirelessinterface of the device.

By way of example, consider environment 1200 of FIG. 12 in which a user1202 has entered a living room with his smart-phone 102-3 in his pocket.Here, assume that the user 1202 has not been in this space before andtherefore the smart-phone 102-2 has no previous context informationassociated with this space. Responsive to sensing an open area orwireless data transmissions of a television 1204, a radar sensor 120 ofthe smart-phone 102-3 begins scanning, through the radar-transparentpocket material, the room and obtaining radar data. As the user changesorientation in the room, the radar sensor 120 continues to scan the roomto obtain additional radar data.

At 1104, 3D radar features are extracted from the radar data. The 3Dradar features may include 3D radar features or a combination of 1D and2D features that are useful to construct 3D radar features. The 3D radarfeatures may include radar reflection features, motion features, orshape features that capture physical aspects of the space or area ofinterest. For example, the 3D radar features may include ranging,position, or shape information for targets in the space, such asfurniture, walls, appliances, floor coverings, architectural features,and the like. In the context of the present example, the context manager112 extracts position and surface radar features of targets in the room,such as the television 1204, plant 1206, door 1208, lamp 1210, picture1212, and sofa 1214. The radar shapes may indicate an approximate shape,surface texture, or position (absolute or relative to other targets) ofeach target in the room.

At 1106, positional data is received from sensors of the device. Thepositional data may include orientation data, inertial data, motiondata, directional data, and the like. In some cases, the positional datais useful to implement synthetic aperture techniques for radar scanningor radar imaging. Alternately or additionally, other sensors of thedevice can provide data indicative of the environment of the space. Forexample, an acoustic sensor may provide data useful to identify anambient noise (e.g., fan noise or machinery hum) present in the space.Continuing the ongoing example, an accelerometer 202 of the smart-phone102-3 provides inertial and orientation data to the sensor fusion engine110 as the user moves throughout the room.

At 1108, a spatial relation of the 3D radar features is determined basedon the positional data. As noted, the positional data can be leveragedto provide a synthetic aperture through which the radar sensor can scanthe area of interest. In other words, as a device moves through a space,the radar sensor can capture physical characteristics of the room asmultiple 3D radar features. The positional data received from the sensorcan then be used to determine spatial relations between the multiple 3Dfeatures or how these features fit together in a 3D space. In thecontext of the present example, the context manager 112 determinesspatial relations between the targets in room by using the inertial andorientation data of the accelerometer 202.

At 1110, a portion of a 3D map is generated based on the 3D radarfeatures and spatial relations thereof. A 3D map can be generated for aportion of the space or room based on landmarks captured by the radarfeatures. These landmarks may include identifiable physicalcharacteristics of the space, such as furniture, basic shape andgeometry of the space, reflectivity of surfaces, and the like. Fromoperation 1110, the method 1100 may return to operation 1102 to generateanother portion of the 3D map of the space or proceed to operation 1112.Continuing the ongoing example and assuming the sensor fusion engine hasscanned most of the room, the context manager 112 generates multipleportions of a 3D map of the room based on the radar features of thetargets 1204 through 1214 in the room and/or overall dimensions of theroom.

At 1112, the portions of the 3D map are combined to create a 3D model ofthe space. In some cases, the portions of the 3D map may be assembled ormeshed by overlapping respective edges. In other cases, the portions ofthe 3D map are combined based on the previously obtained positionaldata. The 3D map of the space may be complete or partial depending on anumber of viable 3D radar features extracted from the radar data. In thecontext of the present example, the context manager 112 meshes thepreviously generated portions to provide a 3D model of the living room.

At 1114, the 3D model of the space is associated with a context of thespace. This can be effective to create a 3D context model of the space.The context can be any suitable type of context, such as a type of room,a security level, privacy level, device operating mode, and the like. Insome cases, the context is user defined, which may include prompting theuser to select from a list of predefined contexts. In other cases,machine learning tools may implement the mapping operations and assign acontext based on physical characteristics of the space. Continuing theongoing example, the context manager 112 associates, based on thepresence of the television 1204 and sofa 1214, a “living room” contextto the space. This context indicates that the area is private, securityrisks are low, and that television 1204 is media capable and can becontrolled through wireless or gesture-driven control functions. Forinstance, media playback on the smart-phone 102-3 may be transferred tothe television 1204 upon entry into the living room.

At 1116, the 3D context model of the space is stored by the device. The3D context model can be stored to local memory or uploaded to the Cloudto enable access by the device or other devices. In some cases, storingthe 3D context model enables subsequent identification of the space viathe radar sensor. For example, the device may maintain a library of 3Dcontext models that enables the device to learn and remember spaces andcontexts associated therewith. Concluding the present example, thecontext manager 112 stores the 3D context model of the living room toenable subsequent access and device configuration, an example of whichis described herein.

FIG. 13 illustrates an example method for configuring context settingsof a device based on a 3D context model, including operations performedby the radar sensor 120, sensor fusion engine 110, or context manager112.

At 1302, a radar sensor of a device is activated to obtain radar datafor a space or area of interest. The radar sensor may be activatedresponsive to movement of the device, such as inertial data or GPS dataindicating that the device is moving into the space or area. In somecases, the radar sensor is activated responsive to detecting knowndevices, such as wireless access points, wireless appliances, or otherwireless devices in the space with which the device has previously beenassociated.

At 1304, 3D radar features are extracted from the radar data. The 3Dradar features may include 3D radar features or a combination of 1D and2D features that are useful to construct 3D radar features. The 3D radarfeatures may include radar reflection features, motion features, orshape features that capture physical aspects of the space or area ofinterest. For example, the 3D radar features may include ranging,position, or shape information for targets in the space, such asfurniture, walls, appliances, floor coverings, architectural features,and the like.

At 1306, positional data is received from sensors of the device. Thepositional data may include orientation data, inertial data, motiondata, directional data, and the like. In some cases, the positional datais useful to implement synthetic aperture techniques for radar scanningor radar imaging. Alternately or additionally, other sensors of thedevice can provide data indicative of the environment of the space. Forexample, an acoustic sensor may provide data useful to identify anambient noise (e.g., fan noise or machinery hum) present in the space.

At 1308, a spatial relation of the 3D radar features is determined basedon the positional data provided by the sensors. As noted, the positionaldata can be leveraged to provide a synthetic aperture through which theradar sensor can scan the area of interest. In other words, as a devicemoves through a space, the radar sensor can capture physicalcharacteristics of the room as multiple 3D radar features. Thepositional data received from the sensor can then be used to determinespatial relations between the multiple 3D features or how these featuresfit together in a 3D space.

At 1310, a set of 3D landmarks of the space is generated based on the 3Dradar features and the spatial orientation thereof These landmarks mayinclude identifiable physical characteristics of the space, such asfurniture, basic shape and geometry of the space, reflectivity ofsurfaces, and the like. For example, 3D landmarks of a conference roommay include a table having legs of a particular shape and an overheadprojector mounted to a mast that protrudes from the ceiling.

At 1312, the set of 3D landmarks is compared to known 3D context models.This can be effective to identify the space in which the device isoperating based on a known 3D context model. In some cases, a match to aknown 3D context model is determined when the set of the 3D landmarkscorrespond to those of the 3D context model. To account for variabilityover time, such as moving or replacing furniture, a match may bedetermined when enough of the 3D landmarks match to meet a predefinedconfidence threshold. In such cases, static 3D landmarks, which mayinclude room geometry and fixed architecture (e.g., staircases), can beweighted heavier to minimize an effect that dynamic landmarks have onmodel matching rates.

At 1314, a context associated with the space is retrieved based on thematching 3D context model. Once a match for the space is determined, acontext to apply to the device can be retrieved or accessed by thedevice. The context may be any suitable type of context, such as aprivacy, meeting, appointment, or security context. Alternately oradditionally, if a context of the 3D context model is incompatible withcurrent device settings or out-of-date, the user may be prompted toselect a context, create a context, or update a context for the space.

At 1316, context settings are configured based on the context associatedwith the space. The context settings configured may include any suitabletype of setting, such as ringer volume, ringer mode, display modes,connectivity to networks or other devices, and the like. Further,security settings or privacy settings, of the device can be configuredto enable or limit the display of secure or private content.

FIG. 14 illustrates an example method for altering context settings inresponse to a change in context, including operations performed by theradar sensor 120, sensor fusion engine 110, or context manager 112.

At 1402, a radar sensor of a device is activated to obtain radar datafor an area of interest. The radar sensor may emit a continuous ordirectional radar field from which signals are reflected by targets inthe area. The targets in the area may include any suitable type ofobject, such as walls, furniture, windows, floor coverings, appliances,room geometry, and the like. By way of example, consider environment1500 of FIG. 15 in which a user 1502 is reading digital contentdisplayed by a tablet computer 102-4. Here, context manager 112activates a radar sensor 120 of the tablet computer 102-4 to obtainradar data for the room in which the table computer is operating.

At 1404, radar features are extracted from the radar data. The 3D radarfeatures may include 3D radar features or a combination of 1D and 2Dfeatures that are useful to construct 3D radar features. The 3D radarfeatures may include radar reflection features, motion features, orshape features that capture physical aspects of the space or area ofinterest. For example, the 3D radar features may include ranging,position, or shape information for targets in the space. In the contextof the present example, a sensor fusion engine 110 of the tabletcomputer 102-4 extracts radar features useful to identify targets withinand a geometry of the living room of environment 1500.

Optionally at 1406, data is received from a sensor of the device. Insome cases, sensor data is useful to determine context for a space. Forexample, an acoustic sensor may provide data associated with anidentifiable ambient noise in the space, such as running water of afountain or a specific frequency of fan noise. In other cases,appliances or electronic devices may emit a particular whine, hum, orultrasonic noise that can be detected by the acoustic sensors to provideidentification data for a particular space.

At 1408, a context for the space is determined based on at least theradar features. In some cases, the context for the device is determinedbased on geometries and occupancies derived from the radar features. Forexample, the context manager may determine a size of the space, numberof other occupants, and distances to those occupants in order to set aprivacy bubble around the device. In other cases, a set of landmarks inthe radar features is compared to known 3D context models. This can beeffective to identify the space in which the device is operating basedon a known 3D context model.

Although described as known, the 3D context models may also be accessedor downloaded to the device, such as based on device location (e.g.,GPS). Alternately or additionally, other types of sensor data can becompared with that of known 3D context models. For example, sounds andwireless networks detected by the device can be compared to acoustic andnetwork data of the known 3D context models. Continuing the ongoingexample, the context manager 112 of the tablet computer 102-4 determinesa context of environment 1500 as “living room,” a private, semi-securecontext.

At 1410, context settings of the device are configured based on thedetermined context. The context settings configured may include anysuitable type of setting, such as ringer volume, ringer mode, displaymodes, connectivity to networks or other devices, and the like. Further,security settings or privacy settings of the device can be configured toenable or limit the display of secure or private content. In the contextof the present example, assume that before an unknown person 1504 entersthe room, the context manager 112 configures display, security, andalerts settings of the tablet computer 102-4 for a private context inwhich these settings are fully enabled or open.

At 1412, the space is monitored for activity via the radar sensor. Theradar sensor may provide a continuous or intermittent radar field fromwhich the reflected signals are received. In some cases, the radarsensor detects activity or targets in the radar field responsive tochanges in the reflected signals. Continuing the ongoing example, theradar sensor 120 monitors environment 1500 for any activity or detectionevents that may indicate a change in context.

At 1414, radar features are extracted from radar data to identify thesource of the activity in the space. This may include extractingdetection, motion, or shape radar features to identify targets in thespace. In some cases, the source of activity is a target leaving thespace, such as someone leaving a room. In other cases, the source ofactivity may include people or objects entering the space. In thecontext of the present example, assume that the unknown person 1504enters the room and approaches the user 1502. In response to this, theradar sensor 120 provides detection and shape radar features 1506 tofacilitate identification of the unknown person 1504.

At 1416, it is determined that the source of the activity changes thecontext of the space. In some cases, other people leaving a spaceincreases user privacy or reduces noise constraints on a device, thusresulting in a more-open context. In other cases, people entering aspace or moving closer to the device can decrease user privacy orincrease security concerns for the device and user. With this reductionin privacy or increased need for security, the context of the device canbecome more private and security oriented. Continuing the ongoingexample, shape radar features 1506 are used in an attempt identify, viafacial recognition, the unknown person 1504. Here, assume that thatfacial recognition fails, and that the context manager 112 determinesthat the presence of an unknown entity changes the context of the spacewith respect to privacy and security.

At 1418, the context settings of the device are altered based on thechange in context of the space. In response to the change in context,the context settings of the device can be altered to compensate for thecontext changes. When the context of the device increases in privacy orsecurity, altering the context settings may include limiting contentexposed by the device, such as by dimming a display, disablingparticular applications, affecting display polarization, limitingwireless connectivity of the device, or reducing audio playback volume.Concluding the present example, in response to detecting a change incontext, the context manager 112 increases privacy and security settingsof the tablet computer 102-4, such as by closing secure applications,reducing volume of alerts and device audio, or reducing a font size ofdisplayed content such that the unknown person would be unable todiscern the tablet's content.

Example Computing System

FIG. 16 illustrates various components of an example computing system1600 that can be implemented as any type of client, server, and/orcomputing device as described with reference to the previous FIGS. 1-15to implement radar-enabled sensor fusion.

The computing system 1600 includes communication devices 1602 thatenable wired and/or wireless communication of device data 1604 (e.g.,received data, data that is being received, data scheduled forbroadcast, data packets of the data, etc.). Device data 1604 or otherdevice content can include configuration settings of the device, mediacontent stored on the device, and/or information associated with a userof the device (e.g., an identity of an actor performing a gesture).Media content stored on the computing system 1600 can include any typeof audio, video, and/or image data. The computing system 1600 includesone or more data inputs 1606 via which any type of data, media content,and/or inputs can be received, such as human utterances, interactionswith a radar field, user-selectable inputs (explicit or implicit),messages, music, television media content, recorded video content, andany other type of audio, video, and/or image data received from anycontent and/or data source.

The computing system 1600 also includes communication interfaces 1608,which can be implemented as any one or more of a serial and/or parallelinterface, a wireless interface, any type of network interface, a modem,and as any other type of communication interface. Communicationinterfaces 1608 provide a connection and/or communication links betweenthe computing system 1600 and a communication network by which otherelectronic, computing, and communication devices communicate data withthe computing system 1600.

The computing system 1600 includes one or more processors 1610 (e.g.,any of microprocessors, controllers, and the like), which processvarious computer-executable instructions to control the operation of thecomputing system 1600 and to enable techniques for, or in which can beembodied, radar-enabled sensor fusion. Alternatively or in addition, thecomputing system 1600 can be implemented with any one or combination ofhardware, firmware, or fixed logic circuitry that is implemented inconnection with processing and control circuits, which are generallyidentified at 1612. Although not shown, the computing system 1600 caninclude a system bus or data transfer system that couples the variouscomponents within the device. A system bus can include any one orcombination of different bus structures, such as a memory bus or memorycontroller, a peripheral bus, a universal serial bus, and/or a processoror local bus that utilizes any of a variety of bus architectures.

The computing system 1600 also includes computer-readable media 1614,such as one or more memory devices that enable persistent and/ornon-transitory data storage (i.e., in contrast to mere signaltransmission), examples of which include random access memory (RAM),non-volatile memory (e.g., any one or more of a read-only memory (ROM),flash memory, EPROM, EEPROM, etc.), and a disk storage device. A diskstorage device may be implemented as any type of magnetic or opticalstorage device, such as a hard disk drive, a recordable and/orrewriteable compact disc (CD), any type of a digital versatile disc(DVD), and the like. The computing system 1600 can also include a massstorage media device (storage media) 1616.

The computer-readable media 1614 provides data storage mechanisms tostore the device data 1604, as well as various device applications 1618and any other types of information and/or data related to operationalaspects of the computing system 1600. For example, an operating system1620 can be maintained as a computer application with thecomputer-readable media 1614 and executed on the processors 1610. Thedevice applications 1618 may include a device manager, such as any formof a control application, software application, signal-processing andcontrol module, code that is native to a particular device, anabstraction module or gesture module and so on. The device applications1618 also include system components, engines, or managers to implementradar-enabled sensor fusion, such as the sensor-based applications 108,sensor fusion engine 110, and context manager 112.

The computing system 1600 may also include, or have access to, one ormore of radar systems or sensors, such as a radar sensor chip 1622having the radar-emitting element 122, a radar-receiving element, andthe antennas 124. While not shown, one or more elements of the sensorfusion engine 110 or context manager 112 may be implemented, in whole orin part, through hardware or firmware.

Conclusion

Although techniques using, and apparatuses including, radar-enabledsensor fusion have been described in language specific to featuresand/or methods, it is to be understood that the subject of the appendedclaims is not necessarily limited to the specific features or methodsdescribed. Rather, the specific features and methods are disclosed asexample ways in which radar-enabled sensor fusion can be implemented.

We claim:
 1. An apparatus comprising: at least one computer processor;at least one radar sensor comprising: at least one radar-emittingelement configured to provide radar fields; and at least oneradar-receiving element configured to receive radar reflection signalscaused by reflections of the radar fields off objects within the radarfields; a plurality of sensors, each of the sensors comprising adistinct type of sensor configured to sense a respective environmentalvariance of the apparatus; and at least one computer-readable storagemedium having instructions stored thereon that, responsive to executionby the computer processor, cause the apparatus to: provide, via theradar-emitting element, a radar field; receive, via the radar-receivingelement, a reflection signal caused by a reflection of the radar fieldoff an object within the radar field; determine, based on the receivedreflection signal, at least one of a radar detection feature, a radarreflection feature, a radar motion feature, a radar position feature, ora radar shape feature of the object; select, based on the determinedradar detection, motion, position, or shape feature of the object, oneof the sensors to provide supplemental sensor data; receive thesupplemental sensor data from the selected sensor; augment the radardetection, motion, position, or shape feature of the object with thesupplemental sensor data to enhance the radar detection, motion,position, or shape feature of the object; and provide the enhanced radardetection, motion, position, or shape feature of the object to asensor-based application effective to improve performance of thesensor-based application.
 2. The apparatus of claim 1, wherein: theinstructions further cause the apparatus to determine an indication of anumber of targets, total reflected energy, moving energy,one-dimensional velocity, one-dimensional velocity dispersion,three-dimensional spatial coordinates, or one-dimensional spatialdispersion based on the reflection signal; and the selecting is furtherbased on the indication.
 3. The apparatus of claim 1, wherein: theinstructions further cause the apparatus to perform a range-Dopplertransform, range profile transform, micro-Doppler transform, I/Qtransform, or spectrogram transform on the reflection signal; and thedetermining is further based on the performed transform.
 4. Theapparatus of claim 1, wherein the sensor-based application comprisesproximity detection, user detection, user tracking, activity detection,facial recognition, breathing detection, or motion signatureidentification.
 5. The apparatus of claim 1, wherein the sensorscomprise two or more of an accelerometer, gyroscope, hall effect sensor,magnetometer, temperature sensor, microphone, capacitive sensor,proximity sensor, ambient light sensor, red-green-blue (RGB) imagesensor, infrared sensor, or depth sensor.
 6. The apparatus of claim 1,wherein augmenting the radar detection, motion, position, or shapefeature of the object with the supplemental sensor data is effective toenhance the radar detection, motion, position, or shape feature of theobject by: increasing a positional accuracy of the radar detection,motion, position, or shape feature of the object; mitigating afalse-positive detection attributed to the radar detection, motion,position, or shape feature of the object; increasing a spatialresolution of the radar detection, motion, position, or shape feature ofthe object; increasing a surface resolution of the radar detection,motion, position, or shape feature of the object; or improving aclassification precision of the radar detection, motion, position, orshape feature of the object.
 7. The apparatus of claim 1, wherein: theradar sensor consumes less power than the selected sensor; and theselected sensor is in a low-power state until the selection.
 8. Theapparatus of claim 1, wherein the apparatus is embodied as asmart-phone, smart-glasses, smart-watch, tablet computer, laptopcomputer, set-top box, smart-appliance, home automation controller, ortelevision.
 9. The apparatus of claim 1, wherein: the radar-receivingelement comprises a plurality of antennas; and the instructions furthercause the apparatus to receive the reflection signal, via the antennas,using beamforming techniques.
 10. A method comprising: providing, via atleast one radar-emitting element of a device, a radar field; receiving,via at least one radar-receiving element of the device, a reflectionsignal caused by a reflection of the radar field off a target within theradar field; abstracting, from the reflection signal, a radar featurethat indicates a physical characteristic of the target; selecting, basedon the radar feature being a first type of radar feature, a first sensorof a plurality of sensors of the device; or selecting, based on theradar feature being a second type of radar feature, a second sensor ofthe sensors of the device that is a different type of sensor than thefirst sensor; receiving supplemental sensor data from the selectedsensor, the supplemental sensor data describing an environmentalvariance corresponding to the distinct type of the selected sensor thatfurther indicates the physical characteristic of the target; augmentingthe radar feature with the supplemental sensor data; and providing theaugmented radar feature to a sensor-based application.
 11. The method ofclaim 10, wherein the first sensor is configured to detect a firstenvironmental variance around the target and the second sensor isconfigured to detect a second environmental variance around the target.12. The method of claim 10, wherein the first or second type of theradar feature comprises a surface feature, motion feature, positionfeature, or detection feature.
 13. The method of claim 10, wherein thesensors of the device comprise at least two of an accelerometer,gyroscope, hall effect sensor, magnetometer, temperature sensor,microphone, capacitive sensor, proximity sensor, ambient light sensor,red-green-blue (RGB) image sensor, infrared sensor, or depth sensor. 14.The method of claim 10, wherein the radar feature indicates a number oftargets in the radar field, total reflected energy, moving energy,one-dimensional velocity, one-dimensional velocity dispersion,three-dimensional spatial coordinates, or one-dimensional spatialdispersion.
 15. At least one computer-readable storage media devicecomprising instructions that, when executed by a processing system,cause the processing system to: cause at least one radar-emittingelement to provide a radar field; receive, via at least oneradar-receiving element, a reflection signal caused by reflection of theradar field off a target within the radar field; resolve the reflectionsignal into at least one of a radar detection feature, a radarreflection feature, a radar motion feature, a radar position feature, ora radar shape feature of the target causing the reflection signal;select, based on the radar detection, reflection, motion, position, orshape feature of the target, at least one sensor to provide supplementalsensor data for the radar detection, reflection, motion, position, orshape feature of the target, the supplemental sensor data indicating atleast one environmental variance around the target; receive, from theselected sensor, the supplemental sensor data indicating theenvironmental variance; augment the radar detection, reflection, motion,position, or shape feature of the target with the supplemental sensordata indicating the environmental variance to provide an enhanced radardetection, reflection, motion, position, or shape feature of the target;and expose the enhanced radar detection, reflection, motion, position,or shape feature of the target to a sensor-based application effectiveto improve performance of the sensor-based application.
 16. Thecomputer-readable storage media device of claim 15, wherein the selectedsensor comprises an accelerometer, gyroscope, hall effect sensor,magnetometer, temperature sensor, microphone, capacitive sensor,proximity sensor, ambient light sensor, red-green-blue (RGB) imagesensor, infrared sensor, or depth sensor.
 17. The computer-readablestorage media device of claim 15, wherein the instructions further causethe processing system to determine an indication of a number of targets,total reflected energy, moving energy, one-dimensional velocity,one-dimensional velocity dispersion, three-dimensional spatialcoordinates, or one-dimensional spatial dispersion based on thereflection signal; and the selecting is further based on the indication.18. The computer-readable storage media device of claim 15, wherein: theinstructions further cause the processing system to perform arange-Doppler transform, range profile transform, micro-Dopplertransform, I/Q transform, or spectrogram transform on the reflectionsignal; and the resolving is further based on the performed transform.19. The computer-readable storage media device of claim 15, wherein thesensor-based application comprises proximity detection, user detection,user tracking, activity detection, facial recognition, breathingdetection, or motion signature identification.
 20. The computer-readablestorage media device of claim 15, wherein augmenting the radardetection, motion, position, or shape feature of the object with thesupplemental sensor data is effective to enhance the radar detection,motion, position, or shape feature of the object by: increasing apositional accuracy of the radar detection, motion, position, or shapefeature of the object; mitigating a false positive detection attributedto the radar detection, motion, position, or shape feature of theobject; increasing a spatial resolution of the radar detection, motion,position, or shape feature of the object; increasing a surfaceresolution of the radar detection, motion, position, or shape feature ofthe object; or improving a classification precision of the radardetection, motion, position, or shape feature of the object.